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\  Abstract 

\ 

Standard  procedures  for  estimating  the  item  parameters  in  IRT 
models  make  no  use  of  auxiliary  information  about  test  items,  such 
as  their  format  or  content,  or  the  skills  they  require  for 
solution.  This  paper  describes  a  framework  for  exploiting  this 
information,  thereby  enhancing  the  precision  and  stability  of  item 
parameter  estimates  and  providing  diagnostic  information  about 
items'  operating  characteristics.  The  principles  are  illustrated 
in  a  context  for  which  a  relatively  simple  approximation  is 
available:  empirical  Bayes  estimation  of  Rasch  item  difficulty 

parameters. 
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Exploiting  Auxiliary  Information  about  Items  in  the 
Estimation  of  Rasch  Item  Difficulty  Parameters 
Two  active  lines  of  research  item  in  response  theory  (IRT) 
incorporate  additional  information  into  the  process  of  parameter 
estimation,  augmenting  that  conveyed  by  item  responses  alone.  One 
line,  motivated  by  statistical  considerations,  uses  Bayesian 
procedures  to  obtain  more  accurate  estimates  of  item  and  examinee 
parameters.  Enhanced  stability  and  lower  mean  squared  errors  can 
be  achieved  by  assuming  exchangeability  over  item  parameters  of  a 
given  type  (e.g.,  difficulty  parameters),  effectively  shrinking 
estimates  toward  their  mean  in  inverse  proportion  to  the  degree  of 
information  available  directly  about  them  (Mislevy,  1986; 
Swaminathan  &  Gifford,  1982,  1985).  A  second  line,  motivated  by 
psychological  considerations,  incorporates  theories  about  specific 
skills  or  subtasks  required  to  answer  an  item  correctly. 
Scheiblechner  (1972)  and  Fischer's  (1973)  Linear  Logistic  Test 
Model  (LLTM)  is  a  prime  example;  Rasch-model  item  difficulty 
parameters  are  cast  as  linear  combinations  of  more  basic 
parameters  that  reflect  the  contributions  of  psychologically 
salient  features  of  each  item. 

The  purpose  of  this  paper  is  to  bring  out  a  confluence  of 
these  two  lines  of  research.  The  idea  is  to  embed  the  LLTM  in  a 
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Bayesian  framework,  maintaining  the  notion  that  item  features  may 
indeed  tell  us  something  about  item  parameters,  but  admitting  they 
may  not  tell  us  everything.  Final  item  parameter  estimates  are  a 
compromise  between  LLTM  estimates,  where  items  with  identical 
features  would  have  identical  estimates,  and  unrestricted  maximum 
likelihood  estimates. 

In  order  to  focus  on  concepts  rather  than  numerical 
procedures,  we  concentrate  on  a  context  for  which  a  relatively 
simple  approximation  is  available.  The  Rasch  IRT  model  for 
dichotomous  items  is  assumed;  a  linear  regression  model  with 
normal,  homoscedastic  residuals  is  posited  for  item  parameters 
given  their  salient  features;  and,  with  what  is  commonly  called  an 
empirical  Bayes  approximation,  final  item  parameter  estimates  are 
calculated  with  maximum  likelihood  estimates  of  the  regression 
model  treated  as  known.  The  result  is  a  simplified  version  of 
Smith's  (1973)  linear  model  with  response-surface  prior 
distributions . 

The  procedures  are  illustrated  with  data  from  a  fractions 
test  for  junior  high  school  students.  Precision  gains  and 
diagnostic  uses  of  the  approach  are  discussed. 
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Background 

This  section  briefly  reviews  the  three  components  of  an  IRT 
model  that  incorporates  auxiliary  information  about  items.  First 
is  the  item  response  model--specifically,  in  this  presentation, 
the  Rasch  model.  Following  that  are  overviews  of  Bayesian 
estimation  of  item  parameters  and  of  the  linear  logistic  test 
model . 

The  Rasch  Model 

Let  x„  denote  the  response  of  examinee  i  to  item  j ,  taking 
the  value  1  if  correct  and  0  if  not.  The  Rasch  model  (Rasch, 
1960/1980)  gives  the  probability  of  a  correct  response  as 


P.(0.)  -  P(x. .  -  1\6 . ,0.) 
j  i  ij  1  i 


-  exp(&i  -  0j)/[l  +  exp(?1  -  0.)]  ,  (1) 

where  0 ^  characterizes  the  difficulty  of  item  j  and  9 ^ 
characterizes  the  ability  of  examinee  i.  Under  the  usual 
assumption  of  local  independence,  the  probability  of  a 
vector  pattern  x^  -  (x^ , . . . , '  of  responses  to  n  items  is 
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P(x  |*  ,£)  -  n  P  (tf  )  iJ  Q  (0.) 

j  J  J 


where  Qj(0)  “  1  -  Pj(0)  and  £  ”  . &r? '  '  AssuminS  the 

independence  of  responses  over  examinees,  the  probability  of  the 

data  matrix  X  -  (x^ . x^) '  of  N  examinees  is  the  product  of 

expressions  like  Equation  2: 

P(X|£,/8)  -  n  P(x  \6  0)  .  (3) 

i 

Once  X  has  been  observed,  Equation  3  is  interpreted  as  a 
likelihood  function,  and  provides  a  basis  for  estimating 
parameters.  The  literature  offers  a  number  of  alternative 
procedures  for  doing  so,  including 

o  joint  maximum  likelihood  (JML) ,  which  finds  values  of  /9 

and  each  9  that,  taken  together,  maximize  Equation  3  (Wright 
&  Panchapakesan,  1969); 

o  conditional  maximum  likelihood  (CML) ,  which  finds  the 
maximizing  value  of  ^  given  examinees'  total  scores 
(Andersen,  1973);  and 
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o  marginal  maximum  likelihood  (MML) ,  which  finds  the  maximizing 
value  of  f)  after  integrating  over  a  distribution  of 
examinee  parameters  (Bock  &  Aitkin,  1981;  Thissen,  1982). 
These  solutions  provide  similar  estimates  of  fi  when  neither  the 
number  of  items  or  examinees  is  small;  under  appropriate 
assumptions  they  are  asymptotically  equivalent,  consistent,  and 
multivariate  normal  (for  details  see  Haberman,  1977,  on  CML  and 
JML,  and  De  Leeuw  &  Verhelst,  1986,  on  CML  and  MML.) 

We  will  have  use  for  the  normal  approximation  to  MML  in  a 
subsequent  section.  The  MML  likelihood  function  is  obtained  from 
Equation  3  by  marginalizing  over  the  examinee  distribution: 

1^(2 IX)  -  n  f  P(xi|5,^)  p(ff)  dff  ,  (4) 

where  p(0),  the  density  function  for  examinee  parameters,  may  be 
specified  a  priori  (as  in  Bock  and  Aitkin,  1981,  and  Thissen, 

1982)  or  estimated  from  the  data  (as  in  Cressie  and  Holland, 

1983) .  When  both  the  numbers  of  items  and  examinees  are  large, 
the  likelihood  function  is  approximately  a  product  over  items  of 
independent  normal  distributions: 
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L^IX)  i  n  expf-(^  -  )2/2«7j  ]  ,  (5) 

A 

where  B  are  MML  estimates  ana  a.  are  their  estimated  standard 
-  : 

errors.  (Large  N  is  sufficient  for  multivariate  normality,  but 
large  n  is  also  necessary  for  independence.) 

Bayesian  Estimation 

The  simultaneous  estimation  of  many  parameters  can  often  be 
improved  when  it  is  reasonable  to  consider  subsets  of  parameters 
as  exchangeable  members  of  corresponding  populations  (Efron  & 
Morris,  1975;  Lindley  &  Smith,  1972).  The  subjective  notion  that 
parameters  are  "in  some  sense  similar"  implies  a  correlational 
structure  on  prior  beliefs,  which  can  be  formalized  by  modeling 
the  parameters  as  if  they  were  a  random  sample  from  a  population 
whose  parameters  are  themselves  imperfectly  known.  Data  related 
directly  to  each  individual  parameter  also  conveys  information 
about  the  higher- level  population  parameters;  the  population 
structure  in  turn  provides  information  about  the  individual 
parameters . 

In  typical  applications,  resulting  estimates  of  individual 
parameters  are  drawn  toward  the  center  of  their  distribution  in 
inverse  proportion  to  the  amount  of  information  available  about 
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them  directly.  An  intuitive  justification  of  shrinkage  is  that 
unrestricted  ML  estimates  contain  sampling  errors,  so  we  would 
expect  that  the  more  extreme  estimates  reflect  in  part  large 
sampling  errors  in  that  direction.  This  reasoning  is  consistent 
with  the  fact  that  the  expected  variance  of  ML  estimates  in  such 
cases  generally  exceeds  the  variance  of  the  true  parameters. 

Swarainathan  and  Gifford  (1982)  applied  this  idea  to  the  Rasch 
model  by  assuming  exchangeability  over  examinees  and  over  items. 

In  a  Bayesian  extension  of  JML,  they  provide  estimation  equations 
for  the  joint  mode  of  fi  and  9  in  the  posterior  distribution 

P<£,g|X)  cc  P(X|£,£)  p(£)  p(£)  ,  (6) 

where  p (9)  and  p(/3 )  are  marginalizations  over  respective  normal 
distributions,  the  parameters  of  which  are  estimated  in  part  from 
the  data.  As  expected,  Swaminathan  and  Gifford's  simulations 
showed  the  Bayesian  estimates  to  be  closer  to  their  overall  mean 
than  unrestricted  maximum  likelihood  estimates,  and  to  have 
smaller  mean  squared  error. 

A  similar  extension  of  MML  is  described  in  Mislevy  (1986). 
Marginalizing  over  6  but  not  over  the  mean  n  and  standard 
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deviation  4>  of  identical  normal  priors  for  the  p's,  he  gives 

2 

estimation  equations  for  the  joint  mode  of  fi ,  p,  and  <f>  in  the 
posterior  distribution 

?(P,H,4>2)  <x  L^(p\X)  x  n  p(/S  \n,4>2)  x  p O,*2)  .  (7) 

j  J 

As  with  Swaminathan  and  Gifford's  procedure,  this  approach  also 
yields  estimates  of  P's  that  are  closer  to  their  estimated  mean 
than  those  of  the  corresponding  maximum  likelihood  procedure. 

The  Linear  Logistic  Test  Model 

In  addition  to  positing  a  Rasch  model  for  item  responses,  as 
in  Equations  1  through  3,  the  LLTM  assumes  a  linear  model  for  the 
item  parameters: 


K 


qkjr?k 


(8) 


or,  in  matrix  notation, 
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£  -  Q'2  • 

The  basic  parameters  of  the  LLTM  are  k  -  1 . K.  They 

reflect  the  additive  contributions  to  item  difficulty  of  selected 
item  features.  The  vector  q^  contains  coefficients  relating  i“em 
j  to  basic  parameters.  In  Fischer's  (1973)  calculus  example,  q 
indicated  the  number  and  the  type  of  operations  a  pupil  must  carry 
out  in  order  to  solve  a  differentiation  item.  In  Mitchell's 
(1983)  analysis  of  Paragraph  Comprehension  subtests  from  the  Armed 
Services  Vocational  Aptitude  Battery,  q  conveyed  semantic  and 
lexicographic  features  of  a  question  and  an  associated  reading 
passage.  The  reader  is  referred  to  Fischer  and  Formann  (1982)  for 
additional  applications  of  the  LTTM. 

Estimates  of  LLTM  basic  parameters  can  be  obtained  by 
suitable  modification  of  JML,  CML,  and  MML  algorithms  for  the 
unconstrained  Rasch  model.  Differences  in  -2  log  likelihood 
between  the  two  models  can  be  compared  with  the  chi-square 
distribution  on  n  -  K  degrees  of  freedom,  to  test  the  significance 
of  the  constraints  of  the  LLTM  under  the  assumption  that  the 
unrestricted  Rasch  model  is  true. 

Fischer  and  Formann  (1982)  note  that  the  initial  hope  of 
explaining  all  reliable  variation  of  item  difficulties  in  terms  of 
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basic  parameters  has  not  been  fulfilled;  rigorous  tests  of  fit 
almost  always  reject  the  LLTM.  This  finding  is  consistent  with 
what  test  developers  have  known  for  decades:  two  items  written  to 
test  the  same  skill  will  differ  in  difficulty  as  a  function  of 
idiosyncratic  features  such  as  visual  format  and  word  choice. 

Typically,  however,  a  meaningful  amount  of  variation  can  be 
explained.  The  proportion  of  variance  of  unconstrained  estimates 
accounted  lor  was  76  percent  in  Fischer's  calculus  test,  and 
ranged  from  66  to  96  percent  in  Mitchell's  Paragraph 

A 

Comprehension  tests.  Even  though  LLTM  estimates  ^  -  Q'r/  are  not 
wholly  acceptable  as  estimates  of  f) ,  then,  their  ability  to 
relate  item  performance  to  cognitive  theory  has  proven  useful  in 
applications  such  as  assessing  treatment  effects  and  modeling  item 
bias.  To  the  extent  that  LLTM  does  fit,  it  aids  an  understanding 
of  just  what  makes  items  difficult.  To  the  extent  that  it  does 
not  fit,  departures  indicate  items  that  are  unexpectedly  hard  or 
easy  given  the  features  that  usually  determine  difficulty.  Poor 
item  construction  or  alternative  response  strategies  can  be 
detected  in  this  way. 
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A  Combined  Model 

Rationale 

The  assumption  of  exchangeability  in  the  Bayesian  estimation 
procedures  described  in  a  preceding  section  typically  leads  to 
item  parameter  estimates  that  are  more  stable  and  have  lower  mean 
squared  errors.  Strictly  speaking,  however,  assuming 
exchangeability  over  all  parameters  of  a  given  type,  and 
consequently  shrinking  them  all  to  the  same  center,  is  justified 
only  if  we  have  no  prior  information  to  distinguish  among  them. 
This  is  rarely  the  case  in  item  parameter  estimation.  In 
vocabulary  tests,  for  example,  we  know  which  words  are  frequently 
used  and  which  ones  are  not;  we  expect  the  familiar  words  to  be 
easier.  In  Fischer's  calculus  test,  we  would  expect  an  item 
demanding  several  differentiation  rules  to  be  more  difficult  than 
one  demanding  only  a  subset  of  the  same  rules. 

As  Fischer  and  Formann  (1982)  point  out,  we  cannot  generally 
expect  a  few  salient  features  to  explain  item  parameters  in  toto. 
We  can,  however,  express  many  of  our  prior  beliefs  in  terms  of 
such  features.  In  particular,  a  model  combining  key  aspects  of 
the  LLTM  and  the  exchangeability  concept  of  Bayesian  estimation 
might  consider  as  exchangeable  only  parameters  of  items  with  the 
same  pedagogically  or  psychologically  relevant  features. 


i 
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Shrinkage  would  then  be  observed  toward  the  center  of  the  subset 
to  which  an  item  belongs- -as  estimated  from  items  of  that  type  and 
possibly  from  other  items  as  well,  if  they  shared  some  features 
with  it.  This  shrinkage  could  quite  possibly  be  in  the  opposite 
direction  from  the  center  of  the  item  set  as  a  whole. 

The  General  Form  of  the  Model 

Let  the  known  (possibly  vector-valued)  quantity  q^.  represent 
auxiliary  information  about  item  j;  let  p(/9|q)  be  the  density 
function  representing  the  distribution  of  p  parameters  for  items 
with  the  same  (generic)  value  of  q.  (The  possibility  that  p(/3|q) 
may  depend  on  unknown  parameters  is  introduced  below.)  The 
posterior  distribution  of  f) ,  given  the  data  X  and  the  auxiliary 
information  Q  -  (q  , ...,q  ),  is  obtained  as 

p(£|x,q)  “  p(£IQ> 

-  n  /  p(x -\e,p)  p (*)  d*  X  n  p(p  |q  )  .  (9) 

i  ~  j  J  -J 

An  implementation  of  Equation  9  inspired  by  the  LTTM  is  to  assume 
a  linear  regression  model  for  p(/3|q)--a  response  -  surf ace  prior,  as 
introduced  by  Smith  (1973)  in  the  context  of  linear  models.  With 
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Q  and  r?  defined  exactly  as  in  the  LLTM,  we  can  approximate  prior 

2 

beliefs  about  item  parameters  as  MVN(Q'»?,^  I).  Considering 

2 

fj  and  <t>  as  additional  unknown  parameters,  the  marginal 
posterior  is  obtained  as 

p(£,r,,/|X,Q)  a  1^  x  ^‘m  n  exp[-(0  -  q'r,)2/2<t>2}  p(rj  ,<t>2)  .  (10) 

j  J 

As  in  the  LLTM,  a  linear  model  based  on  salient  features  gives  the 

central  tendency  of  items  with  the  same  features  ,  namely 

0^  *  Unlike  the  LLTM,  however,  variation  of  true  parameters 

around  these  central  values  is  anticipated. 

Computational  procedures  for  computing  the  posterior  mode  of 
2 

0,  or  of  0,  n,  and  <f>  jointly,  are  readily  obtained  by 
generalizing  the  algorithms  given  in  Mislevy  (1986).  The 
resulting  solutions  can  be  applied  in  the  2-  and  3-parameter 
logistic  models  as  well  as  for  the  Rasch  model.  The  technical 
details  of  this  solution  are  not  central  to  the  present  paper, 
however;  in  order  to  focus  upon  concepts  and  applications,  we  now 
turn  to  a  relatively  simple  computing  approximation  for  the  Rasch 
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A  Computing  Approximation  for  the  Rasch  Model 

This  section  describes  empirical  Bayes  (EB)  estimation  of 

Rasch  item  parameters,  assuming  normal  linear  regression  on 

salient  item  features.  Two  simplifications  are  applied  to  the 

exact  posterior  distribution  given  in  Equation  10.  First,  the 

marginal  likelihood  function  of  /3  is  replaced  by  the  normal 

approximation  given  in  Equation  5.  Second,  MLE's  of  the 

2 

population  parameters  r,  and  are  treated  as  known,  after  they 

A  A 

have  been  estimated  from  MLE's  B,  with  their  standard  errors  a. 

J  J 

treated  as  known.  (It  is  this  use  of  point  estimates  of 
population  parameters  that  is  commonly  associated  with  the  term 
"empirical  Bayes.")  The  resulting  approximation  takes  the 
following  form: 


P(£|X,Q) 


oc  L^glX)  X  p(g|Q) 


«  L^(£|X)  x  ;;  n  p(0  |q  ,q,tf2)p(r7,*2)  dr,  d*2 

j  J  J 


"  2  A  2 
-<*1  -  -(^i  -  ^5 v 

OC  II  exp  [  - hrr - J —  ]  x  n  exp[  -  -  ] 

j  2o?  j  2^ 
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From  this  combination  of  a  likelihood  and  prior  that  are  both 
proportional  to  independent  normal  densities,  independent  normal 
posteriors  follow  (Box  &  Tiao,  1973,  p.  74): 


-<*>,  ■  3,)2 

p(0|X,Q)  oc  n  exp [  - 3 - ) 

j 


where  the  means  and  variances  are  given  by  well-known  formulas: 


A  n A  A  a  A  aO  A0 

3j  -  (a*^  +  3j2)/(^  +  > 


-2  ,A-2  >2,-1 

"j  '  (<7j  +  *  ) 


Computation  thus  proceeds  in  three  steps: 


Unrestricted  maximum  likelihood  estimates  of  item  parameters 
Point  estimates  of  the  regression  parameters 
Final  estimates  of  item  parameters 


’  *  >V  * 
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Step  1:  Unrestricted  maximum  likelihood  estimates  of  item 
parameters 

A 

Rasch  item  parameter  estimates  p and  corresponding  standard 

A 

errors  a can  be  obtained  with  any  of  a  number  of  widely-available 
computer  programs.  Numerical  values  and  small-sample  properties 
of  JML,  CML,  and  MML  estimates  certainly  differ,  but  any  suffice 
for  our  illustrative  purposes.  For  long  tests  and  many  examinees, 
all  support  the  approximation  of  the  marginal  likelihood  as  a 
product  of  independent  normal  distributions,  with  means  given  by 
maximum  likelihood  estimates  and  standard  deviations  given  by  the 
associated  standard  errors. 

Step  2:  Point  estimates  of  the  regression  parameters 

The  regression  structure  for  item  parameters  and  the  normal 
approximation  for  the  marginal  likelihood  lead  to  the  following 
system  of  regression  equations: 


P .  “  P .  +  e  . 
J  J  J 


2  2 

where  (e^,...,en)  -  MVN [0 , diag(o^ , . . . , ] ,  and 
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where  (f^ 


•  ,fn)  -  MVN(0,^2I). 


Taken  together,  they  imply 


A 


q'.  r\  +  h. 
-J-  J 


where  (hj_ . hn)  -  MVN[0,diag(o2  +  <f>2 . a 2  +  ^2)]. 

2 

MLE's  for  r\  and  4>  can  be  obtained  simultaneously  by 
applying  Dempster,  Laird,  and  Rubin's  (1977)  EM  algorithm.  A 
special  case  of  Braun  and  Jones'  (1985)  implementation  was 
employed  for  the  examples  that  appear  in  the  following  section. 

A  A  ^ 

Using  provisional  estimates  rj  and  <f>  ,  the  E-step  computes 
conditional  expectations  of  the  unknown  item  parameters: 


0-  -  E(0  \0  ,a  ,n,4>  ) 


A  A  AqA  A,-  a  a 

-  (f  0.  +  +  o'/-) 


where  0  ^  -  qjtj  is  the  (provisional)  modeled  mean  for  all  items 
with  same  features  as  item  j .  The  M-step  uses  these  results  to 
produce  improved  estimates: 
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2  -  (Q'9>  Vlj 


and 


A2  -  ~ 
n<T  -  0'.0. 

-J-J 

Cycles  of  this  type  are  repeated  until  convergence  is  attained. 

Because  the  distribution  of  the  hypothetical  "complete  data" 

A  2 

,  with  parameters  <f>  and  r) ,  belongs  to  the  exponential 
family  if  a  is  assumed  known,  convergence  to  a  unique  maximum 
is  assured  (Dempster,  Laird,  &  Rubin,  1977). 

Step  3:  Final  estimates  of  item  parameters 

The  posterior  means  and  variances  for  the  0's  that  follow 
from  our  simplifying  assumptions  can  be  calculated  as  in  Equations 
11  and  12.  The  EB  estimate  0.  is  thus  a  weighted  average  of  the 

A 

ML  estimated  0 ^  and  the  regression  estimate  0  ^  .  The  relative 
weights  are  the  precisions  of  the  two  estimates  being  combined, 
implying  that  . . . 

A 

1.  poorly-estimated  0's  shrink  toward  their  predicted  means 

A 

more  strongly  then  well-estimated  0's ; 


-  2'99'2 
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A 

2.  if  all  P's  are  well-estimated  in  comparison  with  the  estimated 
variation  around  their  modeled  means,  little  shrinkage  occurs 

A 

and  p  approaches  P\  and 

A 

3.  if  all  P's  are  poorly- estimated  in  comparison  with  the 

expected  variation  around  their  modeled  means,  much 

shrinkage  occurs  and  p  approaches  p. 

_  2  a  2  A  2 

Posterior  precision,  or  “  °j  +  ^  c^e  sum 

precision  about  p  conveyed  directly  through  the  likelihood 

function  and  that  conveyed  indirectly  through  knowledge  about  item 

features.  By  exploiting  auxiliary  information,  then,  the 

precision  of  item  parameter  estimates  can  be  increased  without  to 

testing  additional  examinees. 

Empirical  Bayes  estimates  are  distinguished  most 

significantly  from  "true"  Bayes  estimates  by  their  failure  to 

2 

account  for  uncertainty  associated  with  rj  and  <f>  .  The  nature 

of  the  consequent  differences  is  to  overstate  the  apparent 

precision  of  the  final  EB  item  parameter  estimates,  while 

affecting  their  values  only  minimally.  The  posterior  variances 

tend  to  be  too  small,  and  the  distributions  should  be  more 

platykurtic,  like  a  t-distr ibution  rather  than  the  normal.  The 

2 

magnitude  of  these  effects  diminishes  as  rj  and  <f>  are  better 


determined  by  the  data.  Larger  N  generally  leads  to  greater 
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ft 


.V 


precision,  but  test  length  n  and  the  matrix  of  cross-products 
Q'Q  are  also  important.  These  influences  affect  the  precision  of 
regression  parameters  and  residual  variance  in  much  the  same 
manner  as  in  standard  regression  analyses. 

A  Numerical  Example 

This  section  applies  EB  estimation  procedures  to  the  20- item 
Fractions  subtest  of  the  California  Achievement  Test  (CAT) ,  Level 
3,  Form  A  (Tiegs  &  Clark,  1970).  The  data  are  Rasch  item 
difficulty  estimates  and  standard  errors,  estimated  from  the 
responses  of  150  sixth-grade  students  with  the  JML  routine  in 
Wright,  Mead,  and  Bell's  (1980)  BICAL  computer  program.  These 
values  appear  in  Table  1,  along  with  a  specification  of  salient 
features  of  each  item.  These  features,  based  on  the  CAT  table  of 
item  specifications,  are  as  follows: 

1.  Addition  (ADD).  The  student  must  solve  an  addition  problem 
involving  one  or  more  fractions  and/or  mixed  numbers. 

2.  Subtraction  (SUB).  The  student  must  solve  a  multiplication 
problem  involving  one  or  more  fractions  and/or  mixed  numbers. 

3.  Multiplication  (MUL) .  The  student  must  solve  a  multiplication 
problem  involving  one  or  more  fractions  and/or  mixed  numbers. 

4.  Division  (DIV) .  The  student  must  solve  a  division  problem 
involving  one  ore  more  fractions  and/or  mixed  number. 
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5.  Common  denominators  (CD).  The  student  must  find  a  common 
denominator  for  two  fractions  with  unlike  denominators. 

6.  Reduction  (RED).  The  student  must  reduce  a  fraction  or  mixed 
number  to  lowest  terms. 

A  sequence  of  three  models  was  fit  to  these  data: 

Model  1:  EB  item  parameter  estimates  were  obtained  under  an 

assumption  of  global  exchangeability.  That  is,  all  items 
were  shrunk  toward  their  common  mean.  The  resulting 
estimates  approximate  the  results  of  Swaminathan  and 
Gifford's  (1982)  procedures. 

Model  2:  EB  estimates  were  obtained  under  the  assumption  of 

exchangeability  among  items  with  the  same  features,  based  on 
Table  1. 

Model  3:  EB  estimates  were  again  obtained,  after  modifying  the 
model  along  lines  suggested  by  an  examination  of  the 
estimates  and  residuals  from  Model  2. 


Insert  Table  1  about  here 


Model  1:  Twenty  Items,  global  exchangeability 

Most  applications  of  EB  estimation  involve  shrinkage  to  the 
common  center  of  the  parameter  set.  This  is  accomplished  in  our 
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framework  by  using  a  vector  of  ones  for  Q.  The  results  of  such  an 
analysis  for  the  CAT  Fractions  test  are  presented  in  Table  2  and 
Figure  1.  The  grand  mean  toward  which  all  estimates  are  shrunk  is 
0.00  (the  result  of  the  scaling  convention  used  in  BICAL) ;  the 

A  A 

estimated  standard  deviation  <f>  of  the  p's  with  a  treated  as  a,  is 
1.71.  This  compares  with  a  standard  deviation  of  1.74  for  the 

A 

P's,  reflecting  the  expectation  that  a  set  of  maximum  likelihood 
estimates  will  be  more  dispersed  than  the  set  of  parameters  they 
estimate.  Accordingly,  under  the  assumption  of  exchangeability 
over  all  items,  the  EB  estimates  shrink  toward  their  common  mean. 


Insert  Table  1  and  Figure  1  about  here 


They  do  not  shrink  very  much,  though.  If  we  define  shrinkage 

A  A 

for  item  j  as  (p^  -  p/)/(P^  -  qjr;),  then  it  is  only  about  2- 
percent  on  the  average.  The  reason  is  that  the  estimated  variance 
of  p,  about  2.92,  is  very  large  compared  to  the  estimation  error 
variance  of  the  individual  item  parameters,  about  .06  on  the 
average.  Information  from  the  likelihood  function  from  a  sample 
size  of  150  is  sufficient  to  overwhelm  the  information  about 


interitem  similarities,  when  the  items  are  as  dissimilar  in 
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Model  2:  Twenty  items,  exchangeability  given  salient  features 

A  second  model  posits  exchangeability  for  items  with  the  same 
CAT  specifications.  The  Q  matrix  in  this  case  consisted  of  the 
columns  of  feature  indicators  given  in  Table  1.  Estimates  of  rj 
and  <t>  are  given  in  Table  3;  item- level  results  are  listed  in  Table 
4  and  illustrated  in  Figure  2. 


Insert  Tables  3  and  4  and  Figure  2  about  here 


The  values  of  the  regression  parameters  ^  shown  in  Table  3 
are  reasonably  consistent  with  expectations.  The  values  for 
addition,  subtraction,  multiplication,  and  addition  can  be 
interpreted  as  values  to  which  items  exhibiting  that  feature  only 
will  be  shrunk.  Addition  and  subtraction  show  lower  (easier) 
values  than  multiplication  and  division.  The  values  for  common 
denominators  and  fraction  reduction  are  both  positive,  indicating 
additional  difficulty  for  an  item  if  this  subskill  is  demanded  in 
order  to  carry  out  the  basic  operation.  The  modeled  mean  for 
straight  addition  items,  for  example,  is  -2.75;  the  mean  for 
addition  items  that  also  require  reduction  is  -2.75  +  1.90,  or 
-.85.  Such  addition  items  are  nearly  as  hard  as  straight  division 
items . 
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The  residual  standard  deviation  <j>  under  Model  2  is  .58,  much 
lower  than  the  comparable  value  of  1.71  in  Model  1  and  closer  to 
the  typical  standard  error  of  about  .3.  EB  item  parameter 
estimates  in  Table  4  thus  exhibit  greater  shrinkage- -9  to  30 
percent.  Now  that  items  within  the  smaller  subsets  over  which 
exchangeability  is  assumed  are  in  fact  more  similar,  the  structure 
contributes  more  information  with  which  to  improve  item  parameter 
estimates.  Average  posterior  precision  increases  by  roughly  25 
percent,  an  amount  equivalent  to  that  attainable  to  testing  about 
40  more  examinees. 

Note  that  estimates  now  shrink  toward  the  appropriate  one  of 
several  predicted  means  rather  than  to  a  single  overall  mean.  One 
item  whose  EB  estimate  moves  away  from  the  overall  mean  is  item  8, 
the  hardest  of  three  straight  subtraction  items.  Even  though  it 
was  easier  than  average  to  begin  with,  the  imposed  exchangeability 
structure  indicates  that  we  would  expect  it  to  be  easy  based  on 
the  tasks  it  presents;  in  this  particular  data  set,  it  may  have 
been  a  bit  harder  than  we  might  expect. 

The  last  column  in  Table  4,  labeled  "standardized 
difference,"  gives  the  distance  of  an  ML  estimate  from  its 
predicted  center,  in  standard  deviation  units: 


Exploiting  Auxiliary  Information 

28 

residual  (easier  than  expected)  for  this  item  is  balanced  by  an 
atypically  large  positive  residual  for  another  item  (17)  with  the 
same  features . 

Further  examination  of  items  with  large  residuals  reveals  two 
items  that  are  noticeably  easier  than  expected  for  the  same 
reason:  while  formally  fractions  items,  both  Item  1  (straight 
addition)  and  6  (straight  subtraction)  require  only  a  whole  number 
operations  with  a  fraction  carried  along.  Failing  to  distinguish 
these  items  from  straight  addition  or  subtraction  items  that 
combine  two  actual  fractions,  Model  2  overpredicts  the  difficulty 
of  Items  1  and  6. 

A  final  anomaly  appears  in  Figure  2,  for  Item  5.  Item  5  is 
one  of  the  harder  items  to  begin  with,  but  the  regression  model 
yields  a  higher-yet  prediction,  much  higher  than  even  the  highest 
ML  estimate  observed.  This  is  the  only  item  requiring  both  the 
common  denominator  and  reduction  skills,  and  the  higher  prediction 
follows  from  the  additivity  of  the  model.  The  unappealing  result 
suggests  an  interaction  of  sorts;  while  two  additional  subskills 
are  required,  it  appears  likely  that  examinees  who  possess  the  CD 
skill  (the  harder  of  the  two)  also  possess  the  RED  skill.  Thus, 
incremental  difficulty  over  straight  addition  when  both  are 
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present  is  not  much  over  that  expected  from  the  common  denominator 
subskill  alone. 

Model  3:  Eighteen  times,  exchangeability  given  salient  features 
The  final  model  illustrated  here  modified  Model  2  in  three 

ways : 

1.  Items  7  and  16,  which  could  be  solved  by  means  of  properties 
of  operations  alone,  are  eliminated  from  further 
consideration. 

2.  A  column  is  added  to  the  Q  matrix  reflecting  a  new  salient 

feature:  WN,  or  whole  numbers  only,  applying  to  Items  1  and  6 

which  require  just  operations  on  whole  numbers  while  a 
fraction  is  carried  along. 

3.  To  reflect  the  interaction  of  CD  and  RED  observed  for  Item  5, 
its  q  value  for  RED  has  been  changed  from  a  1  to  a  zero.  That 
is,  the  difficulty  parameters  of  addition  items  requiring  CD 
and  RED  are  now  considered  exchangeable  with  those  of  items 
requiring  CD,  the  more  difficult  skill,  alone. 

The  data  for  Model  3  are  shown  in  Table  5.  The  results  of 
the  analysis  are  shown  in  Table  6  (regression  parameter 
estimates),  Table  7  (item-level  results),  and  Figure  3  (a  plot  of 
ML,  EB,  and  regression  estimates).  The  revisions  from  model  2 
reduced  the  residual  standard  deviation  substantially,  from  .58  to 
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.23.  This  is  about  the  same  degree  of  precision  as  is  available 
from  the  likelihood,  so  that  EB  estimates  are  roughly  a  50-50 
compromise  between  ML  and  regression  estimates.  Taking  the 
approximate  posterior  variances  at  face  value -- recal 1  that  they 
are  probably  underestimated- -we  would  conclude  that  the  use  of 
auxiliary  information  about  items  yields  an  increase  in  precision 
equivalent  to  doubling  the  size  of  the  sample  of  examinees 


Insert  Table  5,  6,  and  7  about  here 


The  average  magnitude  of  standardized  residuals  is  about  the 

same  as  that  from  Model  2  because  the  denominator  with  which  thev 

,  2 

are  calculated  decreased  when  the  estimate  of  4>  decreased. 

Neither  these  residuals  nor  Figure  3  exhibit  readily  interpretable 
patterns  of  departures  from  the  model. 


Insert  Figure  3  about  here 

As  with  any  mode  1  - f i tt ing  procedure,  the  analysis  that  led  ro 
Model  3  capitalizes  to  some  degree  upon  idiosvnc rat.  i c  features  of 


the  data  at  hand.  Resulting  estimates  of  precision  are  overly 
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optimistic  for  this  reason  in  addition  to  the  expedients  employed 
by  the  estimation  procedure.  Any  serious  attempt  to  model  item 
difficulties  in  the  fractions  domain  would  obviously  require  more 
data  and  more  thought  than  were  needed  simply  to  illustrate 
computational  procedures. 

Discussion 

The  potential  benefits  of  using  auxiliary  information  about 
items  in  item  parameter  estimation  are  increased  precision  and 
diagnostic  capabilities.  In  the  numerical  example  in  the 
preceding  section,  auxiliary  information  contributed  as  much 
information  about  item  parameters  as  the  likelihood  function  did. 
Conditional  on  the  veracity  of  the  assumed  exchangeability 
structure,  then,  precision  was  increased  by  an  amount  equal  to 
that  attainable  by  doubling  the  number  of  examinees.  Diagnostic 
checks  revealed  two  items  that  might  not  be  measuring  the  skills 
intended,  by  offering  items  that  contained  fractions  but  could  be 
solved  without  manipulating  them. 

The  plausibility  of  the  exchangeability  structure  can  also  be 
verified  with  diagnostic  checks.  Two  additional  safeguards  also 
mitigate  the  effects  of  specification  errors  at  this  stage. 

First,  if  the  structure  is  badly  in  error  and  items  assumed 
exchangeable  turn  out  not  be  verv  similar,  shrinkage  will  be 
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minimal  (as  in  Model  1  of  the  example).  Of  course,  minimal 
shrinkage  does  not  necessarily  signal  misspecif ication  or  lack  of 
exchangeability;  all  other  things  being  equal,  shrinkage  decreases 
as  N  increases.  Second,  increasing  the  sample  size  of  examinees 
leads  to  consistent  item  parameter  estimates  even  if  the 
exchangeability  structure  is  flawed. 

The  simplified  computing  approximation  used  in  this  paper 
works  best  for  the  Rasch  model,  where  it  is  needed  least;  even 
fairly  small  sizes  give  reasonably  good  item  parameter  estimates 
there.  The  same  ideas  can  be  applied  more  profitably  to  IRT 
models  with  more  parameters,  each  less  well-determined  by  data 
(e.g.,  the  3-parameter  logistic  model,  and  models  for  multiple- 
category  item  responses) .  The  computational  procedures  for  the 
general  model  are  then  required,  since  it  may  not  be  possible  to 
obtain  finite  unrestricted  ML  estimates  and  their  standard  errors 
No  explicit  averaging  of  ML  and  regression  estimates  can  be 
accomplished  in  those  cases,  and  Bayesian  estimates  must  be 
obtained  directly  from  item  responses. 
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Table  3 


Estimates  of  Regression  Parameters  under  Model  2 


Effect 


Estimate 


1. 

Addition 

-2.75 

2. 

Subtraction 

-2.08 

3. 

Multiplication 

-  .34 

4. 

Division 

-  .61 

5. 

Common  denominators 

3.50 

6. 

Reduction 

1.90 

Standard  deviation  (4>) 

.58 
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Table  4 


Item-Level  Results  from  Model  2 


Standard¬ 

ized 

Shrink-  differ- 


Items 


1 

-3.73 

2 

-2.02 

3 

1.45 

4 

1.16 

5 

1.63 

6 

-2.42 

7 

-3.23 

8 

-1.05 

9 

1.28 

10 

0.30 

11 

-0.41 

12 

-0.80 

13 

2.22 

.31 

-2.75 

0.58 

-3.61  0 

.20 

-2.75 

0.58 

-2.10  0 

.28 

0.75 

0.58 

1.32  0 

.26 

0.75 

0.58 

1.09  0 

.31 

2.65 

0.58 

1.86  0 

.21 

-2.08 

0.58 

-2.38  0 

.27 

-2.08 

0.58 

-3.02  0 

.18 

-2.08 

0.58 

-1.14  0 

.27 

1.42 

0.58 

1.30  0 

.21 

-0.18 

0.58 

0.24  0 

.18 

-0.34 

0.58 

-0.40  0 

.18 

-0.34 

0.58 

-0.76  0 

.38 

1.56 

0.58 

2.02  0 

.31 

1.56 

0.58 

1.68  0 

.28 

1.56 

0.58 

1.44  0 

.18 

-0.61 

0.58 

-1.29  0 

.21 

-0.61 

0.58 

0.16  0 

.27 

1.29 

0.58 

1.28  0 

.28 

1.29 

0.58 

1.39  0 

.25 

1.29 

0.58 

1.09  0 
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Table  5 


Item  Data  and  Salient  Features:  Reduced  Set 


Item 

A 

b 

A 

a 

1 

ADD 

2 

SUB 

3 

MUL 

4 

DIV 

5 

CD 

6 

RED 

7 

WN 

1 

-3.73 

.  31 

1 

0 

0 

0 

0 

0 

1 

2 

-2.02 

.20 

1 

0 

0 

0 

0 

0 

0 

3 

1.45 

.28 

1 

0 

0 

0 

1 

0 

0 

4 

1.16 

.26 

1 

0 

0 

0 

1 

0 

0 

5 

1.63 

.31 

1 

0 

0 

0 

1 

0 

0 

6 

-2.42 

.21 

0 

1 

0 

0 

0 

0 

1 

(7) 

8 

-1.05 

.18 

0 

1 

0 

0 

0 

0 

0 

9 

1.28 

.27 

0 

1 

0 

0 

1 

0 

0 

10 

.30 

.21 

0 

1 

0 

0 

0 

1 

0 

11 

-  .41 

.18 

0 

0 

1 

0 

0 

0 

0 

12 

-  .80 

.18 

0 

0 

1 

0 

0 

0 

0 

13 

2.22 

.38 

0 

0 

1 

0 

0 

1 

0 

14 

1.72 

.31 

0 

0 

1 

0 

0 

1 

0 

15 

1.41 

.28 

0 

0 

1 

0 

0 

1 

0 

(16) 

17 

.26 

.21 

0 

0 

0 

1 

0 

0 

0 

18 

1.28 

.27 

0 

0 

0 

1 

0 

1 

0 

19 

1.41 

.28 

0 

0 

0 

1 

0 

1 

0 

20 

1.05 

.25 

0 

0 

0 

1 

0 

1 

0 

A  'A  A’ 
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Table  6 


Estimates  of  Regression  Parameters  under  Model  3 


Effect 


Estimate 


1. 

Addition 

-1.90 

2. 

Subtraction 

-1.28 

3. 

Multiplication 

-  .32 

4. 

Division 

-  .25 

5. 

Common  denominators 

3.10 

6. 

Reduction 

1.71 

7  . 

Whole  numbers  only 

-1.41 

Standard  deviation  ( <j> ) 

.23 

Exploiting  Auxiliary  Information 


V> 

V 


44 


Table  7 

Item-Level  Results  from  Model  3 


Items 

a 

3 

A 

o 

1 

-3.73 

0.31 

2 

-2.02 

0.20 

3 

1.45 

0.28 

4 

1.16 

0.26 

5 

1.63 

0.31 

6 

-2.42 

0.21 

(7) 

8 

-1.05 

0.18 

9 

1.28 

0.27 

10 

0.30 

0.21 

11 

-0.41 

0.18 

12 

-0.80 

0.18 

13 

2.22 

0.38 

14 

1.72 

0.31 

15 

1.41 

0.28 

(16) 

17 

0.26 

0.21 

18 

1.28 

0.27 

19 

1.41 

0.28 

20 

1.05 

0.25 

A _ & _ 1 


-3. 

,  30 

0. 

23 

-3. 

45 

-1. 

.89 

0. 

.23 

-1. 

96 

1. 

.21 

0. 

23 

1. 

30 

1. 

.21 

0. 

23 

1. 

19 

1. 

,21 

0. 

23 

1. 

35 

-2 

.70 

0. 

.23 

-2. 

.55 

-1 

.28 

0. 

.23 

-1, 

.14 

1. 

.82 

0. 

.23 

1. 

.60 

-0 

.32 

0. 

.23 

0. 

.36 

-0. 

.32 

0. 

.23 

-0. 

.38 

1. 

.39 

0. 

.23 

-0. 

.61 

1. 

.39 

0. 

.23 

1. 

.60 

1. 

.39 

0. 

,23 

1. 

.50 

1. 

,39 

0. 

.23 

1. 

.39 

-0. 

25 

0. 

23 

0. 

02 

1. 

.46 

0. 

23 

1. 

38 

1. 

,  46 

0. 

23 

1. 

44 

1. 

,46 

0. 

23 

1. 

27 

Standard¬ 

ized 

Shrink-  Differ- 


c 

age 

_ ence 

0 

18 

0 

65 

-i 

11 

0 

15 

0 

44 

-0 

43 

0 

18 

0 

61 

0 

67 

0 

17 

0 

57 

-0 

14 

0 

18 

0 

65 

1 

10 

0 

15 

0 

47 

0 

89 

0 

14 

0 

39 

0 

80 

0 

17 

0 

59 

-1 

53 

0 

15 

0 

47 

-0 

42 

0 

14 

0 

39 

-0 

30 

0 

14 

0 

39 

-1 

65 

0 

19 

0 

74 

1 

89 

0 

18 

0 

65 

0 

87 

0 

18 

0 

61 

0 

07 

0 

15 

0 

4  7 

1 

66 

0 

17 

0 

59 

-0 

50 

0 

18 

0 

61 

-0 

13 

0 

17 

0 

55 

-1 

21 

vs 


1937/03/19 


Educational  Testing  Serv i ce/Mi s I evy 


Dr.  Leonard  Fe I dt 
Lindquist  Center 
for  Measurement 
Un i ve rs i ty  of  I oua 
I owa  City,  I A  52242 

Dr.  Richard  L.  Ferguson 
American  College  Testing 
Program 
P.O.  Box  168 
Iowa  City,  IA  52240 

Dr.  Gerhard  Fischer 
Liebiggasse  5/3 
A  1010  Vi enna 
AUSTRIA 

Dr .  Myron  F i sch I 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Prof.  Donald  Fitzgerald 
University  of  New  England 
Department  of  Psychology 
Armidale,  New  South  Wales  2351 
AUSTRALIA 

Mr .  Paul  Foley 

Navy  Personnel  RSD  Center 

San  Diego,  CA  92152-6800 

Dr .  A  I f  red  R.  F reg I y 
AFOSR/NL 

Bolling  AF6,  DC  20332 

Dr.  Robert  D.  Gibbons 
University  of  II  I inois— Chicago 
P.O.  Box  6998 
Chicago,  IL  69680 

Dr .  Jan  ice  G i f  f ord 
University  of  Massachusetts 
School  of  Education 
Amherst,  MA  01003 

Dr.  Robert  Glaser 
Learning  Research 

&  Development  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15260 


Dr.  Bert  Green 
Johns  Hopkins  University 
Department  of  Psychology 
Charles  &  34th  Street 
Baltimore,  MD  21218 

Dipl.  Pad.  Michael  W.  Habon 

Universitat  Dusseldorf 

Erz i ehungsw i ssenschaf t I iches 

Un i vers i tatsstr .  1 

D-4000  Dusseldorf  1 

WEST  GERMANY 

Dr.  Ronald  K.  Hambleton 
Prof,  of  Education  &  Psychology 
University  of  Massachusetts 
at  Amherst 
Hills  House 
Amherst,  MA  01003 

Dr.  Delwyn  Harnisch 
University  of  Illinois 
51  Gerty  Drive 
Champaign,  IL  61820 

Ms.  Rebecca  Hetter 
Navy  Personnel  RSD  Center 
Code  62 

San  Diego,  CA  92152-6800 

Dr.  Pau I  W.  Ho  I  I  and 
Educational  Testing  Service 
Rosedale  Road 
Princeton,  NJ  08541 

Prof.  Lutz  F.  Hornke 
Institut  fur  Psychologie 
RWTH  Aachen 
Jaegerstrasse  17/19 
D-5100  Aachen 
WEST  GERMANY 

Dr.  Paul  Horst 
677  G  Street,  #184 
Chula  Vista,  CA  90010 

Mr.  Dick  Hoshaw 
OP- 135 

Ar I i ngton  Annex 
Room  2834 

Washington,  DC  20350 


1987/03/19 


Educational  Testing  Serv i ce/Mi s I evy 


Or.  Lloyd  Humphreys 
University  of  Illinois 
Department  of  Psychology 
603  East  Daniel  Street 
Champaign,  IL  61820 


Dr .  William  Koch 
University  of  Texas-Austin 
Measurement  and  Evaluation 
Center 

Austin,  TX  78703 


Dr.  Steven  Hunka 
Department  of  Education 
University  of  Alberta 
Edmonton,  Alberta 
CANADA 

Dr.  Huynh  Huynh 
College  of  Education 
Univ.  of  South  Carolina 
Columbia,  SC  23208 

Dr.  Robert  Jannarone 
Department  of  Psychology 
University  of  South  Carolina 
Columbia,  SC  29208 

Dr.  Dennis  E.  Jennings 
Department  of  Statistics 
University  of  Illinois 
1409  West  Green  Street 
Urbana,  IL  61801 

Dr.  Doug  las  H.  Jones 
Thatcher  Jones  Associates 
P.0.  Box  6640 
10  Traf  a  I  gar  Court 
Lawrence v i I  I e ,  NJ  08648 

Dr.  Mil  ton  S.  Katz 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Prof.  John  A.  Keats 
Department  of  Psychology 
University  of  Newcastle 
N.S.W.  2308 
AUSTRALIA 

Dr.  G.  Gage  Kingsbury 

Portland  Public  Schools 

Research  and  Evaluation  Department 

501  North  Dixon  Street 

P.  0.  Box  3107 

Portland,  OR  97209-3107 


Dr.  James  Krantz 
Computer-based  Education 
Research  Laboratory 
University  of  Illinois 
Urbana,  IL  61801 

Dr.  Leonard  Kroeker 
Navy  Personnel  R?«D  Center 
San  Diego,  CA  92152-6800 

Dr .  Dary I  I  Lang 

Navy  Personnel  ROD  Center 

San  Diego,  CA  92152-6800 

Dr.  Thomas  Leonard 
University  of  Wisconsin 
Depa  tment  of  Statistics 
1210  West  Dayton  Street 
Madison,  WI  53705 

Dr.  Mi chae I  Levine 
Educational  Psychology 
210  Educat ion  Bldg. 

Un  i  ve rs i ty  of  Illinois 
Champaign,  IL  61801 

Dr.  Charles  Lewis 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  Robert  L i nn 
College  of  Education 
University  of  Illinois 
Urbana,  IL  61801 

Dr.  Robert  Lockman 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0266 

Dr.  Freder ic  M.  Lord 
Educational  Testing  Service 
Princeton,  NJ  08541 


1987/03/19 


Educational  Testing  Se rv i ce/M i s I e vy 


Dr.  James  Lumsden 
Department  of  Psychology 
University  of  Western  Australia 
Ned  lands  W.A.  6009 
AUSTRALIA 

Dr.  Mil  ton  Mai er 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0268 

Dr .  William  L .  Ma I oy 
Chief  of  Naval  Education 
and  Training 
Nava  I  Air  Station 
Pensacola,  FL  32508 

Dr.  Gary  Marco 
Stop  3 1  — E 

Educational  Testing  Service 
Princeton,  NJ  08451 

Dr,  Clessen  Martin 
Army  Research  Institute 
5001  Eisenhower  Blvd. 
Alexandria,  VA  22333 

Dr.  James  McBride 
Psychological  Corporation 
c/o  Harcourt,  Brace, 

Javanov , ch  Inc . 

1250  West  6th  Street 
San  Diego,  CA  92101 

Dr.  Clarence  McCormick 

HQ,  MEPCOM 

MEPCT-P 

2500  Green  Bay  Road 
North  Chicago,  IL  60064 

Dr.  Robert  McKinley 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr .  James  McM i chae I 
Technical  D i rec to r 
Navy  Personnel  R&D  Center 
San  D i ego ,  CA  92152 


Dr.  Barbara  Means 
Human  Resources 

Research  Organization 
1100  South  Washington 
Alexandria,  VA  22314 

Dr.  Robert  Mi s I evy 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  William  Montague 

NPRDC  Code  13 

San  Diego,  CA  92152-6800 

Ms.  Kathleen  Moreno 
Navy  Personnel  ROD  Center 
Code  62 

San  Diego,  CA  92152-6600 

Headquarters,  Marine  Corps 
Code  MPI-20 
Washington,  DC  20380 

Dr.  W.  Alan  Nicewander 
University  of  Oklahoma 
Department  of  Psychology 
Oklahoma  City,  OK  73069 

Deputy  Technical  Director 

NPRDC  Code  01 A 

San  Diego,  CA  92152-6800 

Director,  Training  Lab o rat o 
NPRDC  (Code  05) 

San  Diego,  CA  92152-6800 

Director,  Manpower  and  Pers 
Laboratory , 

NPRDC  (Code  06) 

San  Diego,  CA  92152-6800 

Director,  Human  Factors 

Z  Organizational  Systems 

NPRDC  (Code  07. 

San  Diego,  CA  92152-6800 

Fleet  Support  Office, 

NPRDC  (Code  301) 

San  Diego,  CA  92152-6800 


Library,  NPRDC 
Code  P201L 

San  Diego,  CA  92152-6800 


1987/03/19 


Educational  Testing  Se rv i ce/M i s I evy 


Commanding  Officer, 

Naval  Research  Laboratory 
Code  2627 

Washington,  DC  20390 

Dr.  Harold  F.  O'Neil,  Jr. 

School  of  Education  -  WPH  801 
Department  of  Educational 
Psychology  &  Technology 
University  of  Southern  California 
Los  Angeles,  CA  90089-0031 

Dr .  James  Olson 
WICAT,  Inc. 

1875  South  State  Street 
Orem,  UT  84057 

Office  of  Naval  Research, 

Code  1142PT 
800  N.  Quincy  Street 
Arlington,  VA  22217-5000 
(6  Copies) 

Office  of  Naval  Research, 

Code  125 

800  N.  Quincy  Street 
Arlington,  VA  22217-5000 

Assistant  for  MPT  Research, 
Development  and  Studies 
OP  0187 

Washington,  DC  20370 

Dr .  Judith  Orasanu 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Jesse  Orlansky 
Institute  for  Defense  Analyses 
1801  N.  Beauregard  St. 

A  I  exandr i a,  VA  2231 1 

Dr.  Randolph  Park 
Army  Research  Institute 
5001  Eisenhower  Blvd. 

Alexandria,  VA  22333 

Wayne  M.  Patience 
American  Council  on  Education 
GED  Testing  Service,  Suite  20 
One  Dupont  Circle,  NW 
Washington,  DC  20036 


Dr .  James  Pau Ison 
Department  of  Psychology 
Portland  State  University 
P.0.  Box  751 
Portland,  OR  97207 

Administrative  Sciences  Department, 
Naval  Postgraduate  School 
Monterey,  CA  93940 

Department  of  Operations  Research, 
Naval  Postgraduate  School 
Monterey,  CA  93340 

Dr .  Mark  D .  Reckase 
ACT 

P.  0.  Box  168 
I owa  City,  IA  52243 

Dr .  Malcolm  Ree 
AFHRL/MP 

Brooks  AFB,  TX  78235 

Dr.  Barry  Riegelhaupt 
HumRRO 

1100  South  Washington  Street 
Alexandria,  VA  22314 

Dr.  Carl  Ross 
CNET-PDCD 
Building  90 

Great  Lakes  NTC,  IL  60088 
Dr.  J.  Ryan 

Department  of  Education 
University  of  South  Carolina 
Co  I umb i a ,  SC  29208 

Dr.  Fumiko  Samejima 
Department  of  Psychology 
University  of  Tennessee 
Knoxv  i  I  I  e  ,  TN  37916 

Mr.  Drew  Sands 

NPRDC  Code  62 

San  Diego,  CA  92152-6800 

Lowe  II  Schoe r 

Psychological  &  Quantitative 
Foundat ions 
College  of  Education 
University  of  I owa 
Iowa  City,  IA  52242 


& 


Educational  Testing  Se rv i ce/M i s I e vy 


Dr.  Mary  Schratz 

Navy  Personnel  PSD  Center 

San  Diego,  CA  92152-6800 

Dr .  Dan  Sega  I  I 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152 

Dr.  W.  Steve  Se I Iman 
OASD (MRA&L) 

28269  The  Pentagon 
Washington,  DC  20301 

Dr.  Kazuo  Shigemasu 
7-9-24  Kugenuma-Ka i gan 
Fujusawa  251 
JAPAN 

Dr .  William  Sims 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0268 

Dr .  H.  Wallace  S i na i ko 
Manpower  Research 

and  Advisory  Services 
Smithsonian  Institution 
801  North  Pitt  Streef 
Alexandria,  VA  22314 

Dr.  Richard  E.  Snuw 
Department  of  Psychology 
Stanford  University 
Stanford,  CA  94306 

Dr.  Richard  Sorensen 
Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Dr.  Paul  Speckman 
University  of  Missouri 
Department  of  Statistics 
Co  I umb i a ,  MO  65201 

Dr.  Judy  Spray 
ACT 

P.0.  Box  168 
I owa  City,  I A  52243 

Dr.  Martha  Stocking 
Educational  Testing  Service 
Princeton,  NJ  08541 


Dr .  Pete r  Sto I  of  f 
Center  for  Naval  Analysis 
200  North  Beauregard  Street 
Alexandria,  VA  22311 

Dr.  William  Stout 
University  of  Illinois 
Department  of  Mathematics 
Urbana,  IL  61801 

Maj .  Bill  Strickland 
AF/MPXOA 
4E168  Pentagon 
Washington,  DC  20330 

Dr.  Hariharan  Swaminathan 
Laboratory  of  Psychometric  s 
Evaluation  Research 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01003 

Mr.  Brad  Sympson 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152-6S00 

Dr.  John  Tangney 
AFOSR/NL 

Bolling  AFB,  DC  20332 

Dr.  Kikumi  Tatsuoka 
CERL 

252  Engineering  Research 
Laboratory 
Urbana,  IL  61801 

Dr.  Maurice  Tatsuoka 
220  Education  Bldg 
1310  S.  Sixth  St . 

Champaign,  IL  61820 

Dr.  Dav id  Th i ssen 
Department  of  Psychology 
University  of  Kansas 
Lawrence,  KS  66044 

Mr.  Gary  Thomasson 
Un  i vers / ty  of  Illinois 
Educational  Psychology 
Champaign,  IL  61820 


Educational  Testing  Serv i ce/M i s I evy 


1987/03/19 


Dr.  Robert  Tsutakawa 
University  of  Missouri 
Department  of  Statistics 
222  Math.  Sciences  Bldg. 

Columbia,  MO  65211 

Dr.  Ledyard  Tucker 
University  of  Illinois 
Department  of  Psychology 
603  E.  Dan i e I  Street 
Champaign,  IL  61820 

Dr.  Vern  W.  Urry 
Personnel  R&D  Center 
Office  of  Personnel  Management 
1900  E.  Street,  NW 
Washington,  DC  20415 

Dr .  Dav id  Vale 
Assessment  Systems  Corp. 

2233  University  Avenue 

Su i te  310 

St.  Paul  ,  MN  55114 

Dr .  F rank  V i c i no 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152-6800 

Dr.  Howard  Wainer 
Division  of  Psycho  I og i ca  I  Studies 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  Ming-Mei  Wang 
Lindquist  Center 
for  Measurement 
Un i vers i ty  of  I owa 
Iowa  City,  IA  52242 

Dr.  Thomas  A.  Warm 
Coast  Guard  Institute 
P.  0.  Substation  18 
Oklahoma  City,  OK  73169 

Dr.  Brian  Waters 
Program  Manager 
Manpower  Analysis  Program 
HumRRO 

1100  S.  Washington  St. 

Alexandria,  VA  22314 


Dr.  David  J.  Weiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MN  55455 

Dr.  Ronald  A.  Weitzman 
NPS,  Code  54Wz 
Monterey,  CA  92152-6800 

Major  John  Welsh 

AFHRL/MOAN 

Brooks  AFB,  TX  78223 

Dr.  Douglas  Wetzel 
Code  12 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Dr .  Rand  R .  Wilcox 
University  of  Southern 
Cal i f orn i a 

Department  of  Psychology 
Los  Angeles,  CA  90007 

German  Military  Representative 
ATTN:  Wolfgang  Wildegrube 
Stre i tkraef teamt 
D-5300  Bonn  2 

4000  Brandywine  Street,  NW 
Washington,  DC  20016 

Dr.  Bruce  Willi ams 
Department  of  Educational 
Psycho  I ogy 

University  of  Illinois 
Urbana,  IL  61801 

Dr .  Hilda  Wing 
Psychological  Corporation 
c/o  Harcourt,  Brace, 

Javano v  i  ch  Inc  . 

1250  West  6th  Street 
San  Diego,  CA  92101 

Dr.  Mart  in  F.  Wi skof f 
Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Mr.  John  H.  Wolfe 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152-6800 


TfWV 


1987/03/19 


Educational  Testing  Serv i ce/Mi s I evy 


Dr.  George  Wong 
Biostatistics  Laboratory 
Memorial  S I oan-Ketter i ng 
Cancer  Center 
1275  York  Avenue 
New  York,  NY  10021 

Or.  Wallace  Wulfeck,  III 
Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Dr.  Kentaro  Yamamoto 
Computet — based  Education 
Research  Laboratory 
University  of  Illinois 
Urbana,  IL  61801 

Dr.  Wendy  Yen 
CTB/McGraw  Hill 
Del  Monte  Research  Park 
Monterey,  CA  93940 

Dr.  Joseph  L.  Young 
Memory  &  Cogn i t i ve 
Processes 

National  Science  Foundation 
Washington,  DC  20550 


'jr*  1 


vi.i  i 


